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Progress  Report 

" Visual  Navigation  &  Space  Perception", 
M.S.  Banks 

During  the  last  two  and  a  half  years,  we  worked  on 
two  general  problems:  Surface  perception  and  heading 
perception.  In  this  progress  report,  we  review  the 
work  leading  up  to  our  current  work  (thus  some  of  the 
material  appeared  on  last  year's  progress  report)  and 
then  we  discuss  the  work  completed  this  past  year. 

1.  Surface  Perception 

The  problem  of  visual  space  perception  is  the 
recovery  of  the  location,  shape,  size,  and  orientation  of 
objects  in  the  environment  from  the  pattern  of  light 
reaching  the  eves.  The  visual  system  uses  disparities 
between  the  two  retinal  images  to  glean  information 
about  the  3-D  layout  of  the  environment.  In  the  past 
five  wars,  we  have  been  investigating  how  disparity  is 
used  to  recover  surface  orientation.  Most  of  the  work 
has  concerned  determining  the  slant  of  an  isolated 
surface  rotated  about  a  vertical  axis.  This  problem  is 
interesting  because  the  pattern  of  disparities  depends 
not  dnlv  on  slant,  but  also  on  location  relative  to  the 
head  (Ogle,  19^0). 

The  first  part  of  this  section  is  basically  the  same  as 
last  war's  progress  report  because  we  need  to  explain 
the  background  to  the  work  we  accomplished  during 
the  two  and  a  half  year  grant  period.  If  you  have 
a  I  read  \  read  this  background  material  from  previous 
progress  reports,  vou  can  skip  ahead  to  page  5. 

Figure  1  depicts  the  geometry  for  binocular  viewing 
ot  ,i  vertical  plane.  The  objective  gaze-normal  surface  is 
the  plane  perpendicular  to  the  cyclopean  line  of  sight. 
The  slant  S  is  the  angle  by  which  the  plane  of  interest  is 
rotated  about  a  vertical  axis  from  the  gaze-normal 
surface. 


Figure  1.  Binocular  viewing  geometry.  See  text. 

What  signals  are  available  for  slant  estimation?  One 
important  signal  is  horizontal  disparity.  For  a  smooth 


surface  slanted  about  a  vertical  axis,  the  horizontal 
disparity  pattern  can  be  represented  locally  by  the 
horizontal  size  ratio  (HSR;  Figure  1),  the  ratio  of 
horizontal  angles  the  patch  subtends  in  the  left  and 
right  eyes  (Rogers  &  Bradshaw,  1993).  Changes  in  HSR 
produce  obvious  and  immediate  changes  in  perceived 
slant,  so  this  signal  must  be  involved  in  slant 
estimation.  However,  HSR  by  itself  is  ambiguous.  To 
illustrate  the  ambiguity.  Figure  2  shows  several  surface 
patches  that  give  rise  to  HSRs  of  1  and  1.04.  For  each 
HSR  value,  there  is  an  infinitude  of  possible  slants 
depending  on  the  surface's  location.  Clearly,  the 
measurement  of  HSR  alone  does  not  allow  an 
unambiguous  estimate  of  the  surface  s  orientation  nor 
do  any  other  descriptions  of  horizontal  disparity 
(Longuet-Higgins,  1982).  A  main  purpose  of  our  work 
has  been  to  determine  what  other  signals  are  used,  in 
combination  with  horizontal  disparity,  by  the  visual 
system  and  to  determine  how  those  signals  are 
combined  to  determine  surface  slant. 


Figure  2.  Ambiguity  of  HSR.  Plan  view  with  the  abscissa 
representing  lateral  position  and  the  ordinate  forward 
position.  The  line  segments  represent  surface  patches  that 
give  rise  to  HSR  =  1  (upper  panel)  and  HSR  =  1.04  (lower 
panel). 

Another  potentially  useful  signal  is  vertical  disparity 
which  can  be  represented  by  the  vertical  size  ratio 
(VSR;  Figure  1),  the  ratio  of  vertical  angles  subtended 
by  a  surface  patch  in  the  left  and  right  eyes.  VSR  varies 
with  the  location  of  a  surface  patch  relative  to  the  head, 
but  does  not  vary  with  surface  slant  (Gillam  & 
Lawergren,  1983).  The  gray  circles  in  Figure  3  show  the 
VSR  at  various  locations  in  the  visual  plane.  Another 


2 


Visual  Navigation  and  Space  Perception,  M.S.  Banks 


signal  is  the  rate  of  change  in  VSR  with  azimuth 
(dVSR/dy);  this  signal  depends  strongly  on  distance 

and  less  so  on  slant. 

Other  useful  signals  are  provided  by  sensed  eye 
position.  Ignoring  torsion,  each  eye  has  one  degree  of 
freedom  in  the  visual  plane.  We  can  thus  represent 
binocular  eye  position  by  two  values,  yand  /i,  the 
version  and  vergence  of  the  eyes  (Figure  1). 


Figure  3.  Im.I'SR  contours.  Plan  view.  Abscissa  represents 
lateral  position  and  ordinate  forward  position.  Each  contour 
represents  the  regions  in  space  for  which  VSR  is  constant; 
each  contour  represents  a  different  VSR. 

Finally,  useful  slant  information  can  be  gleaned 
Irom.  nonstereoscopic  signals  such  as  the  texture 
gradient  created  by  projection  onto  the  retinae  of 
surfaces  with  statistically  regular  textures  (Cutting  & 
Millard.  NS4.  Buckley  &  Frisby,  1993;  Cumming  et  al, 
1SW3).  Such  cues  were  present  in  older  stereoscopic 
work  using  real  objects  (e.g..  Ogle,  1938;  Gillam  et  al, 
19SS).  In  more  recent  work  with  computer  displays, 
there  is  still  generally  a  perspective  cue  that  indicates 
that  the*  surtuce  is  trontoparallel  to  the  head  (e.g., 
Rogers  N  Bradshaw,  1995;  Howard  &  Kaneko,  1994). 
Neither  the  slant  specified  by  a  given  texture  gradient 
nor  the  uncertainty  of  the  estimation  varies  with 
distance  or  azimuth  (Sedgwick,  1986;  Backus  et  al, 
1994| 

An  unambiguous  estimate  of  slant  can  be  obtained 
from  carious  combinations  of  the  above-mentioned 
signals.  For  example,  slant  can  in  principle  be  estimated 
from  HSR  and  sensed  eye  position  (Ogle,  1950;  Foley, 
19S0).  From  Backus  et  al  (1999); 

.V  -  tan  i  -  In  HSR  -  tan  y) .  (1.1) 

H 

The  estimates  of  /i  and  y(fi  and  y)  are  presumably 
derived  from  extra-retinal,  eye-position  signals. 
Correcting  HSR  via  eye  position  has  the  important 
consequence  of  compensating  for  the  changes  in 
binocular  viewing  geometry  that  occur  with  changes  in 
distance  and  azimuth  (Kaneko  &  Howard,  1996;  Ogle, 
1950). 


Slant  can  also  be  estimated  from  retinal-image 
information  alone  (Garding,  et  al.,  1995;  Gillam  & 
Lawergren,  1983;  Koenderirtk  &  van  Doom,  1976; 
Mayhew  &  Longuet-Higgins,  1982).  From  Backus  et  al: 

5  = -tan  ( — In - )  (1.2) 

ft  VSR 

where  ju  can  be  measured  from  retinal  image  properties 
alone.  In  the  terminology  of  Garding  et  al  (1995),  n 
"normalizes"  the  slant  (scales  HSR  for  changes  due  to 
viewing  distance)  and  VSR  "corrects  the  slant 
(corrects  HSR  for  changes  due  to  azimuth). 

In  summary,  certain  subsets  of  signals  allow 
unambiguous  estimation  of  slant  and  we  can 
summarize  them  with  three  calculations  (Banks  & 
Backus,  1998a):  (1)  slant  estimation  from  HSR  and  eye 
position  ( SIISR  Ep ),  (2)  slant  estimation  from  HSR  and 

VSR  (SmKVSK),  and  (3)  slant  estimation  from 

nonstereoscopic  cues  such  as  perspective  (  S,. ). 

In  natural  viewing,  the  slant  estimates  derived  from 
these  three  methods  should  on  average  agree. 
However,  each  signal  measurement  is  subject  to  error, 
so  even  in  natural  viewing,  the  estimates  will  differ. 
Because  a  surface  can  only  have  one  slant  at  a  time,  the 
visual  system  must  derive  one  estimate  from  the  set  of 
somewhat  discrepant  signals.  In  our  conceptualization, 
the  weight  associated  with  each  slant  estimate  is  a 
function  of  its  estimated  reliability,  and  the  estimated 
reliability  is  based  in  turn  on  the  quality  of  the 
information  present  in  the  signals  (e.g.,  Landy  et  al., 
1995;  Heller  &  Trahiotis,  1996).  Several  factors  influence 
signal  reliability.  For  example,  consider  the  effects  of 
increasing  viewing  distance.  As  distance  increases, 
there  is  no  effect  on  the  information  carried  by  the 
perspective  signal  (assuming  broadband  texture; 
Sedgwick,  1986),  but  the  information  carried  by  HSR  is 
reduced  because  a  given  set  of  slants  maps  onto  ever 
smaller  ranges  of  HSR.  Consequently,  nonstereoscopic 
slant  estimates  should  be  weighted  more  heavily 
relative  to  stereoscopic  slant  estimates  as  viewing 
distance  increases;  experimental  evidence  confirms  this 
expectation  (Buckley  &  Frisby,  1993;  Backus  &  Banks, 
1999). 

Some  of  our  experiments  examined  whether  the 
signals  described  above  are  used  in  estimating  slant, 
and  how  the  weights  assigned  to  the  estimates  vary 
across  viewing  conditions  and  stimulus  properties. 

To  do  these  experiments,  we  built  a  haploscope  that 
allows  independent  manipulation  of  eye  position  and 
disparity.  We  examined  the  use  of  the  two  stereoscopic 
means  of  slant  estimation  described  above.  (We  made 
nonstereo,  perspective  information  uninformative  by 
using  a  "back  projection"  procedure;  Banks  &  Backus, 
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1998a.)  Observers  rotated  a  stereoscopic  random-dot 
plane  about  a  vertical  axis  until  it  appeared  normal  to 
the  line  of  sight:  that  is,  they  adjusted  its  slant  until  it 
was  apparently  gaze  normal.  Real  and  simulated 
versions  were  varied  from  15»  to  the  left  of  head¬ 
centric  straight  ahead  to  15*  to  the  right.  Real  version 
was  varied  by  turning  the  haploscope  arms  so  that  the 
observer  rotated  the  eyes  to  the  desired  version 
position.  Simulated  version  was  varied  by  altering  the 
disparity  field.  Thus,  an  observer  might-look  at  a  stereo 
plane  with  eyes  rotated  leftward  while  the  disparities 
presented  were  as  if  the  eyes  were  rotated  rightward.  If 
the  visual  system  relies  on  extra-retinal,  eye-position 
signals  (Sm„,,.,  slant  estimation  by  HSR  and  eye 
position)  in  estimating  the  slant  of  a  stereoscopic 
surface,  then  the  observers'  settings  would  be  predicted 
from  their  actual  eye  positions;  these  predictions  are 
represented  by  the  diagonal  line  in  the  left  panel  of 
Figure  4  If.  on  the  other  hand,  the  system  uses  the 
information  contained  in  the  disparity  field  alone 
,  .s,  ,  estimation  bv  HSR  and  VSR),  the  settings 

would  be  predicted  by  the  simulated  eye  positions; 
these  predictions  are  represented  by  the  three 
horizontal  lines  (one  for  each  of  three  simulated 
eccentricities)  in  the  left  panel  of  Figure  4. 


Version  (deg) 

Figure  I.  ft.-div  nuns  and  results.  Backus  et  ai  (1999).  Natural 
loi;  ni  ii>l<  sellings  is  plotted  as  a  function  of  version.  Left 
panel  Predictions  Slant  estimation  by  HSR  and  eye  position 
predict-  the  diagonal  line.  Estimation  by  HSR  and  VSR 
predict-  the  three  horizontal  lines  (one  for  each  VSR).  Right 
panel.  Results  from  one  of  3  observers.  Squares,  circles,  and 
squares  represent  results  with  different  VSR  values. 

The  results  are  displayed  in  the  right  panel  of  Figure 
4.  The  data  agree  quite  w’ell  with  the  predictions 
of  5  .  The  actual  version  of  the  eyes  had  no  clear 

effect  on  slant  settings  which  is  counter  to  the 
predictions  ot  S(Wf •  Thus,  with  large  targets, 

compensation  for  eccentric  viewing  is  based  primarily 
on  the  pattern  of  horizontal  and  vertical  disparities 
within  the  images  and  little  on  actual  eye  position.  We 


can  summarize  these  findings  by  expressing  the  slant 
estimates  as  weighted  averages  of  the  signals  presented 

to  the  visual  system:  S  =  wHySHSfl  vSR  +  ESHSR  ^  where 
the  w’s  represent  the  associated  weights.  We  can  ignore 
the  nonstereo  slant  estimator  in  this  experiment  (not 
expressed  in  equation)  because  it  always  specified  a 
slant  of  0  and  thereby  could  have  no  influence  in  a 
slant-nulling  task.  The  data  in  Figure  4  can  be  fit  well 
by  this  model  if  vv//  v  =.85  and  wH  t  =.15. 

The  magnitudes  of  vertical  disparities  at  a  given 
azimuth  are  roughly  proportional  to  elevation  above 
the  visual  plane  (VSR  is,  however,  constant  in  the  Fick 
coordinates  we  use  for  our  equations).  Thus,  surfaces 
that  subtend  a  small  vertical  angle  do  not  create  large 
vertical  disparities.  We  took  advantage  of  this  by 
reducing  stimulus  height. 

The  results  for  one  observer  are  shown  in  Figure  5. 
Stimulus  width  was  always  40*,  but  the  height  varied 
from  0-30 »  (left  to  right  in  the  figure).  When  the  height 
was  30*,  we  again  found  that  slant  settings  were 

determined  almost  exclusively  by  S,„K  .  1  lowever,  as 
stimulus  height  was  reduced,  the  slant  settings  became 
more  and  more  consistent  with  5, .  Finally,  with  a 
stimulus  height  of  0»  (horizontal  row  of  dots),  slant 
settings  were  predicted  entirely  byS,nK  ,,  ;  thus,  as  the 
eyes  turned,  different  patterns  of  disparity  were 
required  for  a  gaze-normal  percept. 

These  results  show  clearly  that  the  human  visual 
svstem  employ's  two  means  ot  estimating  slant  of 
stereoscopically  defined  surfaces.  The  weight  given 
S/mw.vR  >s  h>gh  when  the  stimulus  is  large  and  contains 
measurable  vertical  disparities.  The  weight  given 
5mK  H,  is  high  when  the  stimulus  is  short  and  does  not 
contain  readily  measurable  vertical  disparities. 


Figure  5.  Slant  settings  for  different  stimulus  heights.  Natural 
log  of  HSR  is  plotted  in  each  panel  as  a  function  of  version. 
Panels  from  left  to  right  show  data  when  stimulus  height 
varied  from  0-35 •.  Predictions  (see  Figure  4)  are  also  shown 
for  two  means  of  slant  estimation. 

In  the  remainder  of  this  report,  we  will  describe 
slant  estimation  by  linear  combination  of  information 
from  slant  estimators,  each  weighted  in  some 
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statistically  reasonable  fashion.  For  example,  estimated 
slant  about  the  vertical  axis,  can  be  modeled  as: 

S  —  W£lere0S stereo  ^ Nonstereo^ Nonstereo 

where  the  weights  are  positive  and  add  to  1  and 

^  A  A 

^Stereo  =  Wll  ,V$  HSR.VSR  +WH.E^HSR.EP 

The  values  of  the  estimators  will  in  general  not  agree, 
even  in  natural  viewing,  because  the  signals  they  use 
are  subject  to  measurement  error.  Using  linear, 
weighted  combination  to  derive  the  final  estimate  is  an 
example  of  a  weak  fusion  model  (Landy  et  al,  1995; 
Clarke  &  Yuille,  1990).  We  call  it  slant  estimation 
theory. 

The  work  described  above  focused  on  estimation  of 
surface  slant  about  a  vertical  axis.  Naturally,  the  visual 
system  must  estimate  slant  about  any  axis,  not  just  the 
vertical.  One  can  show  that  the  slant  and  tilt  of  a 
smooth  surface  can  be  recovered  locally  from  estimates 
of  the  slant  component  about  the  vertical  axis  (tilt  =  0*; 
Stevens.  198?)  and  the  component  about  the  horizontal 
avis  Hilt  -  90 •)  (Backus  et  al,  1999).  Thus,  we  will 
investigate  slant  estimation  about  the  horizontal  axis 
and  applv  what  we  learn  to  estimation  about  arbitrary 

axes 

Slant  about  the  horizontal  axis  produces  a  vertical 
gradient  of  horizontal  disparity:  so-called  horizontal 
shear  disparity.  We  can  quantify  horizontal  shear 

dispantv  a-  US  It  I-  —  where  the  values  are  defined 

ft:: 

|  u;aie  7  Likewise,  vertical  shear  disparity  is: 
l  |  .  ’’  ,.\<  with  HSR  and  VSR,  we  do  not  claim 

t : 

th.i:  //'>■.  and  \'Sh  are  measured  directly  by  the  visual 
sVsti*m  as  opposed  to  other  quantifications  of  disparity; 
Mther  wr  use  these  quantifications  because  they  are 
t  otn  fim'iit  mathematically. 
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figure  6.  Definition  of  HSh.  Two  eye  s  images  are 
superimposed  (black  =  left  eye;  gray  =  right).  8n  is  the 
horizontal  offset  between  two  corresponding  image  points 
and  p,  is  the  vertical  angle  between  an  image  point  with  no 
disparity  to  a  point  with  shear  disparity. 

The  eves  can  rotate  about  the  lines  of  sight;  these  are 
torsional  movements.  Cycloversion  is  torsional  eye 


movements  in  the  same  direction  and  cyclovergence  is 
torsional  movements  in  opposite  directions.  We  define 
torsional  alignment  as  eye  positions  for  which  the  eyes' 
horizontal  meridia  lie  in  the  same  plane.  When  the  eyes 
are  torsionally  aligned,  slant  about  the  horizontal  axis  is 
given  to  close  approximation  by: 

S  =  -tan''(— lnf/S/i)  (1-3) 

where  p  is  horizontal  vergence  (Banks  et  al,  1999). 
When  the  eyes  are  not  torsionally  aligned,  the  rotation 
of  one  eye  relative  to  the  other  (cyclovergence)  alters 
HSh  (Howard  &  Kaneko,  1994),  so  slant  cannot  be 
estimated  from  Equation  (13).  There  are  two  means  by 
which  the  visual  system  could  compensate  for  the 
change  in  HSh  that  accompanies  cyclovergence. 

1)  Cyclovergence  causes  equal  changes  in  HSh  and 
VSh,  so  correction  could  be  accomplished  by 
comparing  the  two  disparities  (Howard  &  Kaneko, 
1994;  Koenderink  &  van  Doom,  1976).  Banks  et  al 
(1999)  showed  in  this  case  that  slant  ts  to  dose 
approximation: 

5=  -tan'1  (-In—)  0-4) 

M  vsh 

We  refer  to  this  as  slant  estimation  by  HSh  and  VSh 

(  ^  HSh. VSh  )* 

2)  There  may  exist  an  extra-retinal  signal  specifying 
the  eves'  cyclovergence  (Nakayama  &  Balliet,  1977),  so 
correction  could  occur  by  using  that  signal: 

S  =  -  tan'1  (—  In  HSh  -  tan  (O)  (L5) 

M 

where  0)  is  the  eyes'  cyclovergence  (Banks  et  al,  1999). 
We  call  this  slant  estimation  by  HSh  and  eye  position 

(  ^ nsh.hr  )■ 

Usually,  greater  slant  is  perceived  in  stereo-defined 
surfaces  when  slant  is  about  the  horizontal  axis  as 
opposed  to  the  vertical  axis  (Rogers  &  Graham,  1983; 
Mitchison  &  McKee,  1990;  Gillam  &  Ryan,  1992; 
Buckley  &  Frisby,  1993).  Because  the  signals  involved 
are  so  different  for  slant  estimation  about  the  horizontal 
axis  than  for  estimation  about  the  vertical  axis,  there 
are  a  variety  of  possible  explanations  for  this  so-called 
slant  anisotropy.  Comparing  the  results  from  the 
horizontal  axis  experiments  with  our  previous  work 
(e.g..  Backus  et  al,  1999)  will  help  delineate  the  critical 
differences. 

Before  discussing  our  experiments  on  horizontal- 
axis  slant,  we  should  point  out  that  one  can  estimate 
slants  with  respect  to  any  axis  from  the  size  disparities 
(HSR  and  VSR)  and  shear  disparities  ( HSh  and  VSh). 
We  have  shown  that  slant  and  tilt  can  be  recovered  by 
extension  of  Equations  1.2  and  1.4  above.  Specifically, 
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Similar  relationships  can  be  derived  for  horizontal 
disparity  and  sensed  eye  position. 

Now  let  us  return  to  the  discussion  of  slant 
estimation  with  respect  to  the  horizontal  axis.  Howard 
and  Kaneko  (1994)  and  Kaneko  and  Howard  (1996) 
observed  changes  in  perceived  slant  when  HSh  or  VSh 
was  varied;  an  increase  in  HSh  caused  the  stimulus  to 
pitch  top  away  and  an  increase  in  VSh  caused  it  to  pitch 
top  forward,  as  predicted  by  Equation  (1.4).  When  HSh 
was  equal  to  VSh,  the  stimulus  appeared  gaze-normal, 
again  as  predicted  by  Equation  (1.4).  Kaneko  and 
Howard  (1996)  also  looked  for  an  effect  of  eye  position 
(their  Experiment  4),  but  did  not  observe  one. 

!  lowever,  one  would  not  expect  to  observe  an  eye- 
position  effect  in  their  experiment  for  two  reasons, 
l-'irst,  the  stimuli  were  large  random-dot  planes,  so  slant 
estimation  via  vertical  disparity  may  have  dominated 
as  we  observed  in  the  vertical-axis  experiments  (Figure 
m.  Second,  to  stimulate  cyclovergence,  they  presented  a 
r,-so(. .  culorotated  pattern;  cyclovergence  is  slow,  so 
tlii"'  stinuilus  max'  not  have  created  large  enough 
movement^  to  observe  a  perceptual  effect.  They  did 
not  measure  c\ dovergence,  so  one  cannot  assess  this 

po^sihilit\ 

\t  \KVO  in  1<W,  we  presented  data  on  slant 
estimation  about  the  horizontal  axis  (Banks  et  al,  1999). 

•  Ibis  work  is  currently  being  written  up  as  two  papers 
:or  \  Kcscan/i.  tine  paper  (Banks  et  al,  2000) 
contains  the  theoretical  work  and  stereo  experiments 
and  tiie  other  (Hooge  et  al,  2000)  contains  our 
experiments  on  measurements  of  cyclovergence. 

The  goal  in  the  stereo  experiments  w(as  to  vary 
cvclor  ergeiue  and  VSh  independently  to  determine 
whether  the  two  estimation  methods  exist  and,  if  so, 
how  their  outputs  are  combined.  The  experimental 
procedure  is  depicted  in  Figure  7.  We  induced 
cyclovergence  with  a  conditioning  stimulus  composed 
of  horizontal  lines;  the  lines  were  rotated  in  opposite 
directions  in  the  two  eyes.  We  measured  cyclovergence 
response  using  a  nonius  technique.  The  nonius  figure 
(upper  right  panel)  was  flashed  and  the  observer 
judged  whether  the  lines  were  subjectively  parallel.  We 
validated  the  nonius  technique  by  using  3-D  search 
coils  in  van  den  Berg's  lab  in  Rotterdam.  Observers 
performed  the  nonius  task  while  eye  position  was 
measured.  Nonius  and  objective  measures  agreed 
closeiv  (Hooge  et  al,  2000). 


We  conducted  three  stereo  experiments.  In  the  first, 
the  stimulus  used  to  measure  perceived  slant  was  a 
large  random-dot  plane;  the  dots  were  back-projected 
to  render  nonstereo  slant  signals  uninformative. 
Different  amounts  of  VSh  were  added  to  the  stimulus. 
The  plane  was  flashed  and  the  observer  adjusted  HSh 
until  the  plane  appeared  gaze  normal  (lower  right 
panel).  The  procedure  cycled  between  the  conditioning 
stimulus  (2  sec),  nonius  figure  (100  msec),  conditioning 
stimulus  (2  sec),  test  stimulus  (100  msec),  and  so  forth 
until  the  observer  was  satisfied  with  both  settings.  By 
using  this  procedure,  we  knew  the  eyes'  cyclovergence. 


Figure  7.  Experimental  procedure.  Black  lines  represent  left 
eye's  image  and  gray  lines  right  eye's  image.  Conditioning 
stimulus  (left)  is  presented  to  induce  cyclovergence.  Nonius 
technique  (upper  right)  is  used  to  measure  cyclovergence; 
observers  adjust  orientation  of  upper  line  until  subjectively 
parallel  to  lower.  Gaze-normal  task  (lower  right)  is  used  to 
measure  slant  percepts.  Observers  adjust  HSh  until  random- 
dot  plane  appears  gaze  normal. 

Predictions  for  the  gaze-normal  task  are  shown  in 
the  left  panel  of  Figure  8.  Cyclovergence  response  is 
plotted  on  the  abscissa  and  HSh  (at  the  screen)  on  the 
ordinate.  (For  convenience,  HSh  is  expressed  in 
equivalent  degrees  of  cyclorotation.)  If  no 
compensation  occurred  for  changes  in  cyclovergence, 
gaze-normal  settings  would  be  predicted  by  Equation 
(1.3);  the  data  would  lie  on  the  diagonal  line.  If 
complete  compensation  occurred  based  on  VSh 
(Equation  1.4),  the  data  would  lie  on  the  five  horizontal 
lines.  If  complete  compensation  occurred  based  on  eye- 
position  signals  (Equation  1.5),  data  would  lie  on  the 
horizontal  line  at  HSh  =  0. 

Data  from  one  of  the  three  observers  are  shown  in 
the  right  panel  of  Figure  8.  The  data  are  consistent  with 
slant  estimation  by  HSh  and  VSh  except  when  VSh  was 
opposite  in  sign  from  cyclovergence  (upper  left  and 
lower  right  parts  of  the  figure).  We  cannot  determine 
whether  those  deviations  reflect  the  contribution  of 
eye-position  signals  or  a  failure  to  compensate  because 
VSh  was  so  large  in  those  cases. 
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Figure  8.  Predictions  and  results.  HSh  at  the  screen 
(e\ pressed  in  deg)  is  plotted  as  a  function  of  the 

cyclovergence  response.  Left  panel:  Predicted  HSh  settings 
for  the  three  hypotheses.  Right  panel:  Individual  settings  for 
observer  111  1  From  top  to  bottom,  VSh  (in  deg)  was  4,  2,  0,  - 
2  .ip.. !  -t  deg.  Different  symbols  represent  the  different 


Experiments  2  and  3  were  designed  to  determine 
which  is  the  better  account.  In  Experiment  2,  we 
reduced  the  diameter  of  the  random-dot  plane  to  5  deg; 
tins  made  VSh  difficult  to  measure.  In  Experiment  3, 
the  stimulus  was  a  single  smooth  vertical  line;  Lhis 
inuke-.  \‘<h  impossible  to  measure  because  there  are  no 
verttcalh  separated  features.  In  both  cases,  we  found  a 
complete  failure  to  compensate  for  the  eyes' 
i  \  elo\  efitonco.  in  other  words,  when  we  induced 
i  \  c!o\  ergence  changes,  the  observer  saw-  different 
slants  m  the  stimulus.  It  appears  then  that 
compensation  lor  cyclovergence  is  mediated  only  by 
use  ot  vertical  disparities.  We  found  no  evidence  for 
use  ot  an  extra-retinal  signal. 

1  Kinne  the  past  year,  we  completed  two  other 
experiments  and  manuscripts  concerning  surface 
perception  t  >ne  involved  an  analysis  of  binocular  slant 
contrast  <\ an  F.e  et  al,  1999)  and  the  other  involved  use 
ot  haptic  feedback  to  adjust  the  weights  given  to 
ditteren!  slant  estimators  (Ernst  et  al,  2000).  We  also 
completed  data  collection  and  analysis  on  an 
experiment  on  real-world  slant  perception.  The 
manuscript  is  in  draft  form  (James  et  al,  2000). 

We  begin  by  describing  our  analysis  of  slant 
contrast.  There  are  several  stereoscopic  phenomena  in 
which  the  perceived  slant  or  curvature  of  a  surface 
patch  is  not  what  one  would  predict  from  the 
disparities.  The  presence  of  these  illusions  has  been 
taken  as  evidence  that  visual  system  does  not  interpret 
retinal  disparity  accurately  and  thus  does  not  derive 
from  stereopsis  an  accurate  3D  representation  of  the 
scene  (e.g.,  Poggio,  et  al,  1988).  Furthermore, 
explanations  for  the  illusions  have  included  special 


assumptions  about  the  form  of  disparity-encoding 
mechanisms  (Rogers  &  Graham,  1983),  internal  biases 
(Gogel,  1956),  reference  frames  (van  Ee  &  Erkelens, 
1996),  and  other  assumptions.  We  believe  that  slant 
estimation  theory  can  account  for  many  of  these 
illusions;  that  is,  they  might  be  understood  by 
analyzing  the  available  signals  and  their  statistical 
reliabilities  without  assumptions  about  underlying 
mechanisms.  We  propose  analyzing  some  well  known 
stereoscopic  illusions  and  conducting  experiments  to 
determine  if  the  analyses  are  valid. 

One  such  illusion  is  the  slant-contrast  effect 
illustrated  in  Figure  9.  When  cross-fused,  the  cross- 
hatched  planes  create  the  HSR  associated  with  a  plane 
rotated  about  a  vertical  axis  (left  side  far).  The  strip  of 
dots  is  identical  in  the  two  eyes,  so  it  crea'es  the  HSR 
associated  with  a  frontal  plane.  The  dot  strip,  however, 
appears  rotated  in  the  direction  opposite  from  the 
specified  slant  of  the  cross-hatched  planes.  The  illusory 
slant  is  the  slant-contrast  effect  (van  Ee  et  al,  1999). 


Figure  9.  The  slant-contrast  effect.  Cross  fuse.  The  images  of 
the  strip  of  dots  are  identical  in  the  two  eyes,  but  the  strip 
appears  rotated  left  side  near. 

Werner  (1937,  1938)  first  described  the  effect  and 
pointed  out  that  the  perceived  slant  of  the  surrounding 
stimulus  (the  inducer)  was  less  than  the  slant  specified 
by  its  disparity  gradient  and  that  the  perceived  slant  of 
the  internal  stimulus  (the  test  strip)  was  opposite  to  the 
disparity-specified  slant  of  the  inducer.  Since  then  slant 
contrast  has  been  reported  for  a  wide  variety  of  stimuli 
(Ogle,  1946;  Harker,  1962;  Pastore,  1964;  Pastore  & 
Terwilliger,  1966;  Nelson,  1977;  Graham  &  Rogers, 
1982;  Stevens  &  Brookes,  1988;  Fahle  &  Westheimer, 
1988;  Brookes  &  Stevens,  1989;  Howard  et  al,  1993;  van 
Ee,  1995;  van  Ee  &  Erkelens,  1996a;  Pierce  et  al,  1998). 

There  are  many  explanations  for  slant  contrast 
(Howard  &  Rogers,  1995;  van  Ee  et  al,  1999);  they 
involve  special  assumptions  that,  as  we  will  show,  may 
not  be  necessary. 

We  adopted  slant  estimation  theory  for  the  analysis 
of  slant  contrast  (van  Ee  et  al,  1999).  Recall  that  this 
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theory  incorporates  all  the  signals  presented  in  the 
stimulus  and  models  the  resulting  slant  percepts  from 
linear  combination  of  slant  estimators  weighted  by 
their  assumed  statistical  reliabilities.  To  see  how  it 
explains  slant  contrast,  consider  first  the  two  parts  of 
the  stimulus  in  isolation.  The  physical  slants  of  the 
inducer  and  test  strip  are  5,  and  5, .  The  perceived 
slants  of  the  isolated  inducer  and  isolated  test  strip  are 
given  by: 

=  W,J,d+W^lp 

^t.aloie  =  Wid  ^ id  +  Wlp^'P  (1-7) 

Subscripts  I  and  t  refer  to  the  inducer  and  test  strip,  and 
d  and  p  to  stereo  and  nonstereo  (disparity  and 
perspective).  The  weights  are  positive  and  add  to  1. 
The  stereo  estimates  ( Sld  and  S,d)  are  based  on  the 
gradients  of  horizontal  disparity  created  by  the  test 
strip  and  inducer  (plus  other  signals  correcting  for 
distance  and  azimuth).  Similarly,  the  nonstereo 
estimates  are  based  on  nonstereo  perspective  cues 
created  bv  the  strip  and  inducer. 

The  process  of  estimating  the  inducer  and  test  strip 
slants  from  the  signals  they  create  is  depicted  in  Figure 
10.  For  the  test  strip  in  isolation,  the  stereo-  and 
nonstereo-specified  slants  are  both  zero,  so  the  slant 
estimators  on  average  specify  slants  of  zero  and  the 
final  slant  estimate  is  on  average  zero.  For  the  inducer 
in  isolation,  the  stereo-specified  slant  and  nonstereo 
slant  differ:  the  former  is  varied  and  the  latter  is  always 
zero.  The  final  slant  estimate  is  a  compromise  between 
the  stereo  and  nonstereo-specified  slants. 


\  ^i,a/one 


Figure  10.  Slant  estimation  with  the  slant-contrast  stimulus. 
The  inducer's  nonstereo-specified  slant  is  inconsistent  with 
its  stereo-specified  slant.  The  upper  panels,  from  left  to  right, 
depict  slant  estimates  from,  respectively,  test-strip  signals 
alone,  inducer  signals  alone,  and  signals  between  the  two. 
The  lower  panel  depicts  the  final  slant  estimates  for  the  strip 
and  inducer. 
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We  also  need  to  consider  signals  created  jointly  by 
the  two  parts  of  the  stimulus.  ("relative  slant")  is 
the  difference  between  the  slants  of  the  two  surfaces, 
specified  by  disparity.  Snl  is  based  on  the  disparity 
gradient  between  the  two  surfaces  (plus  Other  signals 
needed  to  correct  for  distance  and  azimuth). 

If  all  three  stereo  estimators  (Sld,Sld,  and  Srtl)  are 
unbiased,  it  will  on  average  be  the  case  that: 

$,=$,+5*.  0-8) 

In  the  natural  environment,  signals  created  by  two 
surfaces  in  isolation  (equations  1.6  &  1.7)  and  by  the 
signal  created  between  the  surfaces  (equation  1.8)  are 
on  average  consistent  with  one  another.  In  the  slant- 
contrast  stimulus,  they  are  not.  To  see  how,  first 
consider  the  signals  from  the  surfaces  in  isolation 
(equations  1.6  &  1.7):  the  estimate  for  the  inducer  in 

isolation  (Sialone)  will  be  different  from  both  its  stereo- 

based  and  nonstereo-based  estimates:  5,  JJi(V  *  StJ  *  Slf, , 

Thus,  the  slant  difference  specified  by  the  isolated 
signals  is  not  the  same  as  that  specified  by  the  joint 
signal.  Slant  estimation  theory  states  that  the  visual 
system  reconciles  the  disagreement  by  combining 
information  from  the  available  signals  and  weighting 
each  information  source  according  to  its  estimated 
reliability.  For  the  test  strip: 

S.  n-9) 

The  first  term  is  based  on  signals  from  the  test  strip 
alone  and  the  second  term  on  signals  created  by  the 
inducer  alone  and  the  relative  disparity  gradient 
between  the  inducer  and  test  strip. 

The  inducer's  slant  estimate  is  in  principle  subject  to 
the  same  constraints-the  test  strip  inducing  slant  in  the 
inducer-but,  by  its  construction,  the  slant-contrast 
stimulus  specifies  the  inducer's  slant  much  more 
reliably  than  the  strip's  slant  (van  Ee  ct  al,  1999).  Thus, 
for  the  stimulus  in  Figure  9,  we  can  ignore  the  strip's 
effect  on  the  inducer. 

The  relative  reliabilities  of  the  slant  estimates 
derived  from  the  first  and  second  terms  in  Equation 
(1.9)  determine  their  weights.  The  second  estimator  will 

be  no  more  reliable  than  its  least  reliable  term;  if  Si  alone 

and  are  uncorrelated,  their  reciprocal  reliabilities 
(variances)  will  add. 

We  can  now  predict  the  apparent  slant  of  the  test 
strip.  From  equations  1.6-1 .9,  wid  =  l- wfp  (by 
definition),  and  the  assumption  of  unbiased  estimators: 

s,  =  +  W,t5,d  -W,p(.vi<(  -sip)l .  (1.10) 

If  the  indirect  estimate  of  test  strip  slant  is  much  more 
reliable  than  the  direct  measure,  then  w,  =  0 . 
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Furthermore,  in  the  slant-contrast  stimulus,  S,d  =  Slp  = 
0,  so 

■s,=-Vw.  (L11) 

The  predicted  perceived  slant  of  the  test  strip  is  thus 
opposite  to  the  inducer's  stereo-specified  slant  ( Sld ) 
and  is  proportional  to  the  weight  of  the  inducer's 
nonstereo-specified  slant  (%).  We  showed 
experimentally  that  this  equation  predicts  slant  contrast 
effects  accurately  (van  Ee  et  al,  1999). 

According  to  slant  estimation  theory,  slant  contrast 
is  a  byproduct  of  the  visual  system  s  attempt  to 
reconcile  the  inconsistency  between  the  stereo-  and 
nonstereo-based  signals  produced  by  the  inducer.  It 
follows  that  slant  contrast  would  not  be  observed  if  the 
inconsistency  were  eliminated  by  making  the  inducer's 
stereo  and  nonstereo  signals  consistent:  in  Equation 
1.10,  the  term  weighted  by  w,„  becomes  zero.  We 
confirmed  this  prediction  once  all  inducer  signals, 
including  those  arising  from  the  surface  of  the  CRT, 
were  consistent  (van  Ee,  et  al,  1999).  Notice  that  the 
prevailing  theory  of  slant  contrast  (Rogers  &  Graham, 
19S a;  van  Lie  &  Erkelens,  1996;  Howard  &  Rogers,  1995) 
would  predict  no  effect  of  altering  the  inducer's 
nonstereo-specified  slant. 

During  the  past  year,  we  also  examined  the 
question  ot  how  the  visual  system  determines  how 
much  v.  eight  tii  give  to  one  means  of  slant  estimation 
as  opposed  to  another.  Specifically,  we  examined  how 
the  weights  given  disparity  and  texture  cues  are 
determined  There  are  at  least  three  possibilities.  1) 
Fixed  weights:  the  weights  are  fixed  for  a  given 
situation  and  individual  and  not  subject  to  change 
through  teedback.  2)  Comparison  with  other 
estimators  the  weights  are  learned  over  time  bv 
comparing  a  given  estimator's  output  with  those  of 
other  estimators  and  with  feedback  from  motor 
behavior,  3i  On-line  determination:  the  weights  are 
determined  directly  trom  measuring  the  statistics  of 
estimator  output;  for  example,  if  the  output  of  one 
estimator  tluctuates  less  over  time  than  that  of  another 
estimator,  the  w  eight  of  the  former  should  be. increased 
relative  to  the  weight  of  the  latter. 

The  second  and  third  possibilities  are  not  mutually 
exclusive:  b>oth  means  of  weight  determination  could 
co-exist  in  a  given  estimator. 

We  conducted  an  experiment  with  Marc  Ernst,  a 
Physics  graduate  student,  that  was  designed  to 
examine  the  second  of  the  above  hypotheses. 
Specifically,  we  asked  whether  visuomotor  feedback 
could  affect  the  weights  given  to  stereo  and  nonstereo 
slant  estimators. 

The  experimental- setup  is  schematized  in  Figure  11. 
The  visual  display  was  viewed  in  a  mirror  placed 


above  the  hand.  The  index  finger  and  thumb  of  the 
right  hand  were  placed  in  separate  haptic  feedback 
devices  (depicted  beneath  the  mirror).  The  Phantom 
feedback  devices  provided  force  to  the  finger  and 
thumb  and  thereby  simulated  a  virtual  surface. 


Figure  11.  The  experimental  setup  for  the  haptic  feedback 
experiment.  The  visual  stimulus  is  presented  in  the  display 
screen  above  and  viewed  in  the  mirror  placed  above  the 
hand.  The  index  finger  and  thumb  of  the  right  hand  are 
placed  in  Phantom  haptic  feedback  devices.  The  observer 
touches  and  grasps  virtual  objects  in  the  workspace.  Haptic 
feedback  creates  the  sensation  of  touching  a  real  object  or 
surface. 

The  experiment  had  three  phases  which  are 
depicted  in  Figure  12:  1)  pre-adaptation,  2)  adaptation, 
and  3)  post-adaptation.  The  pre-  and  post-adaptation 
phases  were  purely  visual  tasks.  The  visual  stimulus  in 
those  phases  was  a  plane  slanted  about  a  vertical  axis. 
Its  slant  was  specified  in  two  ways:  1)  Nonstereo 
signals  consisting  of  the  texture  gradient  and  outline 
shape  and  2)  stereo  signals  consisting  of  geometrically 
correct  disparity  gradients.  The  nonstereo-  and  stereo- 
specified  slants  differed  by  ±30,  ±20,  or  ±10  deg. 
Observers  adjusted  the  slant  of  the  perceived  surface 
until  it  appeared  gaze  normal;  the  adjustment  affected 
the  nonstereo-  and  stereo-specified  slants  equally 
thereby  holding  the  difference  between  them  constant. 
Although  no  feedback  was  given  during  the  pre-  and 
post-adaptation  phases,  observers  had  no  difficulty 
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performing  this  task.  They  were  also  unaware  of  the 
conflict  between  the  slant  signals. _ t 
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Figure  12.  Experimental  design  for  haptic  feedback 
experiment.  The  three  lines  in  the  left  and  right  panels 
represent  the  stereo-and  nonstereo-specified  slants  as  well  as 
the  slant  percept.  See  text  for  details. 

During  the  adaptation  phase,  the  Phantom  provided 
haptic  feedback.  Observers  pushed  a  small  visible  cube 
with  the  index  finger  to  targets  on  a  visible  slanted 
plane.  Haptic  feedback  was  given  according  to  the 
nonstereo-specified  and  not  the  stereo-specified  slant. 
The  difference  between  the  nonstereo  and  stereo  slants 
was  random  during  adaptation,  so  haptic  feedback  was 
only  correlated  with  the  nonstereo-specified  slant.  The 
adaptation  phase  lasted  30-45  minutes.  Observers  were 
questioned  after  the  experiment:  they  were  not  aware 
that  the  haptic  feedback  was  unusual  in  any  way. 

Tlu*  results  are  shown  in  Figure  13  which  plots  the 
slant  settings  from  the  pre-  and  post-adaptation  phases. 
The  nonstereo-specified  slant  of  the  surface  when  it 
appeared  ga/e  normal  is  plotted  as  a  function  of  the 
dtlierence  between  the  nonstereo-  and  stereo-specified 
slants  B\  performing  linear  regression  on  these  data, 
we  can  determine  the  weights  given  nonstereo-  and 
stereo-specitied  slants.  The  average  stereo  weight  was 
0.70  during  pre-adaptation  and  0.55  during  post¬ 
adaptation  Although  this  is  a  small  change,  all  eight 
observers  exhibited  the  decrease,  so  the  effect  is  highly 
signiticant  statistically.  This  suggests  that  haptic 
feedback  during  the  adaptation  phase  produced  a 
downweighting  of  the  stereo  slant  estimator.  There 
have  been  numerous  demonstrations  that  changing  the 
relationship  between  vision  and  haptic  feedback  can 
affect  motor  behavior  (reviewed  by  Welch,  1986  and 
Harris,  1980),  but  we  believe  this  is  the  first 
demonstration  that  haptic  feedback  affects  judgments 
in  a  purelv  visual  task. 

We  also  tested  some  alternative  explanations  for 
these  data.  Perhaps  the  decrease  in  stereo  weight 
resulted  from  increased  experience  with  the  visual 
displays  and  not  from  the  low  correlation  between 
haptic  feedback  and  nonstereo-specified  slant  during 
the  adaptation  phase.  We  can  rule  this  out  because  we 


observed  no  decrease  in  stereo  weight  when  nonstereo- 
and  stereo-specified  slants  were  identical  during  the 
adaptation  phase.  Perhaps  the  visual  experience  of 
seeing  a  cube  move  along  a  nonstereo-defined  surface 
and  not  along  a  stereo-defined  surface  causes  a  change 
in  weight.  To  test  this  possibility,  we  recorded  the 
sequence  of  visual  images  presented  during  the 
adaptation  phase  of  each  observer  in  the  main 
experiment  and  showed  those  sequences  to  control 
subjects.  Although  they  experienced  the  same  sequence 
of  images,  they  did  not  exhibit  a  change  in  stereo 
weight. 


Difference  between  Stereo 
&  Nonstereo  Slants  (deg) 

Figure  13.  Results  from  haptic  feedback  experiment.  Slant 
settings  are  shown  for  the  pre-  and  post-adaptation  phrases. 
The.  nonstereo-specified  slant  of  the  visual  stimulus,  when  it 
appeared  gaze  normal,  is  plotted  as  a  function  ol  the 
difference  between  the  nonstereo-  and  stereo-specified  slants. 
The  data  points  are  averages  from  8  observers'  settings. 

We  have  shown  that  haptic  feedback  affects  surface 
perception  in  a  purely  visual  task.  Specifically,  giving 
haptic  feedback  consistent  with  one  slant  estimator  has 
the  effect  of  increasing  the  weight  assigned  to  that 
estimator.  This  observation  is  consistent  with  the  view 
that  calibration  of  spatial  vision  occurs,  at  least  in  part, 
as  a  consequence  of  motor  interaction  with  the 
environment. 

This  work  appeared  recently  in  Nature  Neuroscience 
(Ernst  et  al,  2000).  It  has  received  quite  a  bit  of 
attention  in  the  media  as  well.  It  was  written  up  in  the 
southern  German  newspaper:  Suddeutsche  Zeitung:  (the 
article  was  entitled  "Der  Mensch  sieht  auch  mit  den 
Handen"  which  means  that  people  see  with  their 
hands).  It  is  posted  on  a  news  bulletin  on  the  web: 
http:/ /www.mpg.de/ priOO/ pri6  00.html  and  Dr. 
Ernst  was  interviewed  about  this  study  on  a  local  radio 
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station:  "Hessischer  Rundfunk".  In  addition.  Dr.  Ernst 
won  the  Attempto  Preis  for  this  work.  This  prize  is  an 
annual  award  given  to  the  best  work  in  neuroscience  in 
the  southern  German  area. 

Very  recently,  we  have  completed  an  investigation 
into  the  perception  of  slant  with  real-world  objects.  In 
collaboration  with  Fiona  James  and  Tutis  Vilis  of  the 
University  of  Western  Ontario,  we  examined  how 
people  take  eye  position  into  account  when  asked  to 
judge  the  slant  of  a  surface  in  world  coordinates.  To 
judge  slant  relative  to  the  world,  the  nervous  system 
must  measure  surface  slant  relative  to  the  line  of  sight 
(oculocentric  slant),  eye  position  relative  to  the  head, 
and  head  position  relative  to  the  world  coordinates. 
We  showed  two  things:  1)  people  are  quite  good  at 
judging  object  slant  in  world  coordinates  and  2)  their 
errors  are  the  outcome  of  errors  in  all  three 
measurements.  This  work  was  reported  at  ARVO  this 
war  (lames  et  al,  2000)  and  is  currently  being  written 
up  tor  publication  in  Vision  Research. 

2.  Heading  Perception 

We  have  also  continued  our  research  on  the 
perception  of  self-motion.  During  the  past  two  years, 
five  publications  appeared  from  this  project:  Crowell  ct 
al  (h'OS).  Ehrlich  et  al  (1998),  Freeman  and  Banks 
ipiosi.  Freeman  (1999),  and  Freeman  et  al  (2000).  In 
addition,  we  have  completed  the  theoretical  analysis 
and  prelimmarv  experiments  on  another  project  which 
was  reported  this  year  at  ARVO  (Sibigtroth  et  al,  2000). 
U  e  aKo  completed  construction  of  our  3-axis  rotating 
chair  in  which  we  are  conducting  visual-vestibular 
research  that  is  relevant  to  spatial  disorientation  in 
d\  ution 

As  m  tin-  previous  section,  we  will  first  review  the 
background  material  tor  this  research  project  before 
moving  onto  the  particular  experiments  and  analyses 
that  w ere  completed.  Much  of  this  background  section 
also  appeared  in  last  year's  progress  report,  so  if  you 
read  it  pre\  lousiv,  vou  may  want  to  skip  ahead  to  page 
11.  ' 

As  a  person  moves  through  the  natural 
environment,  images  move  across  the  retina,  the  eyes 
move  relative  to  the  head,  the  head  turns  relative  to  the 
bod  v,  and  the  body  translates  and  rotates  through 
space.  Despite  this  complex  of  various  motions,  the 
nervous  svstem  produces  a  coherent  percept  of  the 
person's  motion  relative  to  environmental  landmarks. 
From  this  percept,  the  human  observer  is  able  to  move 
toward  targets,  avoid  obstacles,  and  guide  complicated 
perceptual-motor  behavior.  We  have  been  examining 
how  the  nervous  system  accomplishes  this.  Our  work 


has  examined  the  analysis  of  visual  signals,  eye- 
velocity  signals,  head-velocity  signals,  and  more.  We 
begin  this  part  of  the  report  by  describing  work  we  did 
on  how  the  visual  takes  the  motion  of  the  retinal  image 
and  the  motion  of  the  eye  in  the  head  into  account 
when  judging  the  velocity  of  an  object. 

If  the  body  and  head  are  stationary,  the  head-centric 
velocity  of  an  object  (H)  is  the  sum  of  retino-centric 
velocity  (R)  and  eye  velocity  (P).  Perceived  head¬ 
centric  velocity  will  be  affected  by  errors  in  the  visual 
system’s  measurements  of  R  or  P.  We  assume  that  the 
perceived  velocity  ( H  )  is  the  sum  of  estimated  retino- 
centric  velocity  (R)  and  estimated  eye  velocity  ( P ). 
Assuming  that  P  is  linearly  related  to  eye  speed,  we 
have:  P  =  eP ,  where  e  is  the  extra-ictinal  gain  factor 

relating  actual  to  estimated  eye  speed.  R  is  the 
estimated  retinal  image  velocity,  so  making  the  linear 
assumption:  R  -  r(Q)R  f  where  r  is  the  retinal  gain 
relating  actual  to  estimated  retino-centric  speed  which 
is  affected  by  a  variety  of  stimulus  properties  (Q) 
including  spatial  frequency,  contrast,  and  size.  It  is 
surprising  that  variations  in  retinal  gain  have  not  been 
incorporated  in  most  previous  models  of  head-centric 
velocity  perception.  We  assume  a  single  value  for  r  for 
each  value  of  Q.  Our  model  of  perceived  head-centric 
velocity  is,  therefore: 

H  =r(Q)//  +  P[e-r(Q)  |.  (2.1) 

r  and  e  are  unknown  properties  of  the  visual  system; 
the  other  variables  are  physical  quantities.  The 
parameters  of  this  model  were  measured  in  speed¬ 
matching  experiments  with  gratings  of  different  spatial 
frequencies  (Freeman  &  Banks,  1998). 

When  an  observer  makes  a  pursuit  eye  movement, 
while  being  presented  a  target  stationary  with  respect 
to  the  head,  the  Filehne  illusion  occurs:  The  target 
appears  to  move  opposite  to  the  eye  movement.  The 
conventional  explanation  for  the  illusion  is  that  the  gain 
of  the  extra-retinal,  eye-velocity  signal  (e)  is  less  than  1, 
so  it  under-estimates  actual  eye  speed  during  pursuit 
movements  (Mack,  &  Herman,  1973,  1978;  Yasui,  & 
Young,  1975);  'the  implicit  assumption  is  that  the  retinal 
gain  (r)  is  1.  We  examined  this  classic  illusion  in  the 
context  of  our  model. 

When  the  perceived  speeds  of  two  stimuli  are  equal, 
Ht  =  H s ,  where  the  subscripts  refer  to  the  test  and 
standard  stimuli.  With  this  equality.  Equation  (2.1)  can 
be  rewritten: 

(2.2) 

r(/,l  H, 

where  r(fs )  and  r(fT)  are  the  retinal  gains  for  the 
standard  spatial  frequency  and  the  test  spatial 
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frequency,  respectively.  Equation  (2.2)  suggests  that 
the  magnitude  of  the  Filehne  illusion  ought  to  depend 
on  the  spatial  frequency  of  the  target.  The  Filehne 
illusion  is  usually  measured  by  setting  the  target's 
speed  such  that  it  appears  stationary  with  respect  to  the 
head.  In  this  case,  Hs  =  0,  so  Equation  (2.2)  becomes: 


Ht=P[  1- 


(2.3) 


where  e  =  r^T -  and  e= — - — .  For  a  given  pursuit 
r(fs)  r(fs ) 

speed,  P  is  constant  and  we  assume  that  e  is  constant 
as  well.  When  ?(fT)>e,  equation  (2.3)  predicts  that 
//,  must  have  the  same  sign  as  P  for  the  target  to 
appear  stationary;  this  prediction  is  consistent  with  the 
classical  literature.  However,  when  r(fT)<e,  which 
could  occur  at  low  spatial  frequencies  where  retinal 
gain  is  low,  the  equation  predicts  that  Hr  must  be 
opposite  in  sign  from  P  for  the  target  to  appear 
stationary.  This  prediction  is  opposite  to  the 
conventional  Filehne  illusion.  There  is  one  report  of  a 
reversed  Filehne  illusion  (Wertheim,  1987).  We  were 
able  to  confirm  the  predictions  of  Equation  (2.3)  quite 
accurately  across  an  8-fold  range  of  spatial  frequencies. 
Specifically \  we  observed  the  reverse  Filehne  illusion  at 
the  spatial  irequencies  predicted  by  the  model. 

We  a  ho  examined  the  Aubert-Fleischl  illusion  in 
which  a  moving  object  appears  to  move  more  slowly 
when  it  is  tracked  with  a  pursuit  eye  movement  than 
when  it  I-  not  tracked.  As  with  the  Filehne,  the  Aubert- 
Heischi  illusion  has  been  assumed  to  result  from  an 
extra-retinal  gain  (e)  less  than  1;  the  implicit 
assumption  again  is  that  the  retinal  gain  (r)  is  1.  Again 
the  mode!  predicts  that  the  Aubert-Fleischl  illusion 
should  reverse  at  low  spatial  frequencies  and  we  were 
able  to  lonlirm  this  experimentally.  Figure  14 
summarizes  the  results  of  this  experiment  (details  in 
the  caption) 

Our  model  and  data  make  clear  that  the  perception 
ot  head  centric  speed  is  affected  by  errors  in  estimating 
speed  ot  eve  movement  and  by  errors  in  estimating 
speed  ot  retinal-image  motion  (Freeman  &  Banks, 
1998).  The  latter  finding  is  particularly  important  for 
understanding  effects  of  lowpass  filtering  and  contrast 
reduction  that  occur  with  visual  aids  such  as  night- 
vision  goggles. 

We  also  applied  what  we  learned  in  the  above  study 
to  the  problem  of  estimating  heading  during  gaze 
rotations.  In  particular,  we  examined  the  timing  and 
gain  of  extra-retinal,  eye-velocity  signals  while  people 
perform  a  self-motion  estimation  task  (Freeman,  1999; 
Freeman,  Banks,  &  Crowell,  2000).  Here  we  describe 
that  work  verv  briefly. 


Non-PursuedGratingSp  eed(°/s) 


Figure  14.  Aubert-Fleischl  illusion  as  a  function  ot  spatial 
frequency.  Observers  reported  whether  a  pursued  or  non- 
pursued  grating  moved  faster  relative  to  the  head.  The 
panels  plot  the  percentage  of  responses  that  the  non-pursued 
grating  appeared  faster  as  a  function  of  the  non-pursued 
grating's  speed.  Eye  pursuit  speed  was  6.2deg/s  (vertical 
dashed  line).  Filled  squares  represent  data  when  the 
frequency  of  pursued  and  non-pursued  gratings  was  1  cpd. 
Filled  circles  represent  data  when  the  gratings'  trequency 
was  0.125  cpd.  Open  circles  represent  the  data  when  the 
frequencies  of  pursued  and  non-pursued  gratings  were  0. 1 2r> 
and  1  cpd,  respectively. 

To  judge  stimulus  motion  relative  to  the  head,  the 
visual  system  must  correct  for  any  eye  movement  the 
occurs  by  using  an  extra-retinal,  eye-velocity  signal. 
Such  correction  is  important  in  a  variety  of  motion 
estimation  tasks  including  judgements  of  object  motion 
relative  to  the  head  and  judgements  of  self-motion 
direction  from  optic  flow.  Speed  and  timing  errors 
were  investigated  using  sinusoidal  pursuit  eye 
movements.  We  described  a  new  illusion — the  slalom 
illusion — in  which  the  perceived  direction  of  self- 
motion  oscillates  left  and  right  when  the  eyes  moves 
sinusoidally.  The  linear  model  of  Freeman  and  Banks 
(1998)  was  used  to  determine  the  gain  ratio  and  phase 
difference  of  extra-retinal  and  retinal  signals 
accompanying  the  Filehne  illusion  and  slalom  illusion. 
The  speed  mismatch  and  timing  differences  were 
measured  in  the  Filehne  and  self-motion  situations 
using  a  motion-nulling  procedure.  Timing  errors  were 
very  small  for  the  Filehne  and  slalom  illusions. 
However,  the  ratios  of  retinal  gain  and  extra-retinal 
gain  were  consistently  less  than  one,  so  both  illusions 
are  the  consequence  of  a  mismatch  between  estimates 
of  retinal  and  extra-retinal  speed. 

The  results  are  quite  relevant  to  aviation  because 
this  work  shows  that  errors  in  self-motion  estimation 
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can  occur  during  pursuit  eye  movements  and  that 
those  errors  are  dependent  on  the  type  of  visual 
stimulus  presented  to  the  pilot.  The  linear  model  allows 
us  to  make  reasonably  general  statements  about  the 
relationship  between  stimulus  parameters  and 
misperceptions  of  self-motion  during  eye  movements. 

We  also  continued  our  work  on  the  use  of  various 
signals  to  estimate  the  direction  of  self-motion.  The 
problem  we  examined  in  the  rotation  problem,  so  we 
begin  with  a  description  of  that  problem,  followed  by  a 
brief  literature  review,  and  then  by  a  description  of  our 
work  during  the  grant  period. 

As  we  move  through  the  environment,  the  retinal 
image  of  that  environment  changes  in  predictable 
ways.  For  example,  if  we  move  in  a  straight  line  our 
self-motion  produces  a  radial  pattern  of  motion  in  the 
retinal  image,  like  that  in  Figure  15A.  The  center  or 
focus  of  the  radial  expansion  (marked  by  a  '+'  in  Figure 
15)  corresponds  to  our  direction  of  motion  (Gibson,  et 
al,  1955).  Re-creating  this  pattern  of  retinal-image 
motion  by  viewing  a  film  or  computer  display 
depicting  our  forward  motion  can  cause  a  compelling 
sensation  that  we  are  in  fact  moving  forward  (Howard, 
1982),  and  under  a  variety  of  conditions  we  can 
accurately  estimate  where  we  are  going  in  the 
simulated  scene  (Warren  et  al,  1988;  Royden  et  al, 
1992) 

W  hen  we  smoothly  shift  gaze  direction  by  turning 
the  e\e  or  head  (e.g.  to  look  at  a  moving  object  or  a 
stationar\  object  to  the  side)  while  still  moving  in  a 
straight  line,  the  pattern  of  retinal-image  motion  is 
more  complex  (Figure  15B).  We  can  re-create  this  type 
o!  retinal  motion  pattern  by  having  observers  hold  the 
eves  still  while  viewing  a  display  that  simulates  both 
their  forward  motion  and  an  eye  movement.  In  this 
case,  observers  report  that  they  are  moving  along  a 
curved  trajectory  (as  though  turning  a  car  while 
looking  torward  through  the  windshield)  rather  than 
along  the'  depicted  linear  path.  When  they  are  asked  to 
adjust  the  position  of  a  marker  in  front  of  them  until  it 
appears  to  sit  upon  their  future  path,  their  responses 
are  strongly  biased  in  the  direction  of  the  perceived 
path  curvature  (Royden  et  al,  1992,  1994;  Banks  et  al, 
1996,  van  den  Berg,  1996).  On  the  other  hand,  self- 
motion  judgments  are  quite  accurate  when  the  identical 
pattern  of  retinal  image  motion  is  created  by  having 
observers  view  a  display  like  that  in  Figure  15A  while 
turning  the  eve  to  pursue  a  target  that  moves  across  it 
(Royden  et  al,  1992,  1994;  Banks  et  al,  1996;  van  den 
Berg,  1996).  Observers  typically  report  that  they 
appear  to  be  moving  on  a  straight  rather  than  a  curved 
path  (Royden,  1994).  In  this  case,  the  observer's  visual 
system  has  extra-retinal  information  about  the  eye 
movement,  probably. consisting  mainly  of  an  efference 


copy  of  the  motor  command  to  turn  the  eye  (Howard, 
1982).  The  visual  system  uses  this  information  to 
compensate  for  the  effects  of  the  eye  movement  on  the 
retinal  motion  pattern;  previous  research  using  self- 
motion  judgments  indicates  that  this  compensation  is 
nearly  complete. 


A)  Linear  Self-motion 
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Figure  15.  Retinal  flow  fields  for  two  viewing  situations.  A) 
Forward  translation  without  a  gaze  rotation.  Observer  is 
fixating  in  constant  direction  and  heading  toward  the  cross. 
B)  Forward  translation  while  making  a  gaze  rotation. 
Observer  is  making  a  rightward  eye  movement. 

During  the  grant  period,  we  examined  the 
contribution  of  depth  information  to  estimating  the 
direction  of  self  motion.  This  work  appeared  in  a  paper 
by  Ehrlich  and  colleagues  in  Vision  Research  in  1998. 
Here  we  describe  that  work  briefly. 

When  presented  with  random-dot  displays  with 
little  depth  information,  observers  cannot  determine 
their  direction  of  self-motion  accurately  in  the  presence 
of  rotational  flow  without  appropriate  extra-retinal 
information  (Royden,  et  al  1994).  On  theoretical 
grounds,  one  might  expect  improved  performance 
when  depth  information  is  added  to  the  display  (van 
den  Berg  &  Brenner,  1994a).  We  examined  this 
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possibility  by  having  observers  indicate  perceived  self- 
motion  paths  when  the  amount  of  depth  information 
was  varied.  When  stereoscopic  cues  and  a  variety  of 
monocular  depth  cues  were  added,  observers  still 
misperceived  the  depicted  self-motion  when  the 
rotational  flow  in  the  display  was  not  accompanied  by 
an  appropriate  extra-retinal,  eye-velocity  signal. 
Specifically,  they  perceived  curved  self-motion  paths 
with  the  curvature  in  the  direction  of  the  simulated  eye 
rotation.  The  distance  to  the  response  marker  was 
crucial  to  the  objective  measurement  of  this 
misperception.  When  the  marker  distance  was  small, 
the  observers'  settings  were  reasonably  accurate  despite 
the  misperception  of  the  depicted  self-motion.  When 
the  marker  distance  was  large,  the  settings  exhibited 
the  errors  reported  previously  by  Royden  et  al  (1994). 
The  path  judgment  errors  observers  make  during 
simulated  gaze  rotations  appear  to  be  the  result  of 
misattributing  path-independent  rotation  to  self- 
motion  along  a  circular  path  with  path-dependent 
rotation.  An  analysis  of  the  information  an  observer 
could  use  to  avoid  such  errors  reveals  that  the  addition 
of  depth  information  is  of  little  use. 

During  the  grant  period,  we  completed  a  project  on 
how  the  visual  compensates  for  head  rotations  in 
estimating  self-motion;  the  manuscript  appeared  in 
Nit tui':'  Newv<aence  (Crowell,  et  al,  1998)  and  was  also 
re\  iewed  tavorably.  in  Science  Neius. 

In  our  previous  work,  we  examined  perception  of 
sell-motion  during  gaze  rotations  caused  by  a  smooth 
pursuit  eve  movement.  Of  course,  gaze  rotations  in 
everyJav  perception,  including  aviation,  occur  due  to 
rotations  ot  the  eye,  head,  and  body  (and  various 
combinations  ot  those).  We  examined  human  ability  to 
estimate  sell-motion  paths  during  gaze  rotations 
caused  In  head  and  body  rotations.  Figure  16  depicts 
the  three  sorts  of  self-motion  conditions  presented  in 
this  experiment.  In  each  case,  the  depicted  self-motion 
path  was  forward  over  a  ground  plane  with  a  path- 
independent  rotation  (about  a  vertical  axis).  The 
simulated  rotation  condition  is  the  same  as  used 
previously  by  Royden  et  al  (1994)  and  Banks  et  al 
(1996).  In  the  head-rotation  condition,  observers  made 
an  active  head  rotation  (about  a  vertical  axis). 
Observers  fixated  a  moving  point  such  that  no  rotation 
of  the  eye  relative  to  the  head  was  required  to  maintain 
gaze  direction  on  the  moving  fixation  point.  Observers 
were  trained  to  make  smooth  head  movements  of  the 
correct  velocity;  during  the  experiment  itself,  the  head 
movements  were  measured  and  trials  were  thrown  out 
when  the  movement  did'  not  meet  pre-established 
velocity  and  position  criteria.  In  the  body-rotation 
condition,  observers  were  rotated  passively  in  a 
vestibular  chair;  again  they  fixated  a  moving  point  such 


that  no  rotation  of  the  eye  relative  to  the  head  occurred 
and  such  that  no  rotation  of  the  head  relative  to  the 
body  occurred.  (This  experimental  condition  was 
conducted  in  Richard  Andersen's  laboratory'  at 
Caltech).  The  retinal  images  were  identical  across  the 
three  gaze-rotation  conditions,  so  differences  in 
performance  must  manifest  differences  in  the  usage  of 
different  extra-retinal  signals.  In  all  conditions, 
observers  made  judgments  of  the  perceived  self-motion 
path. 


Figure  16.  Gaze  rotation  conditions:  Simulated  ga/e  rotation 
(no  rotation  of  eye,  head,  or  body),  head  rotation  (rotation  of 
head  relative  to  the  body),  and  body  rotation  (rotation  of  the 
body  relative  to  the  world).  The  retinal  images  are  identical 
across  the  three  conditions. 

The  results  from  one  of  six  observers  are  displayed 
in  Figure  17.  The  unfilled  circles  in  the  left  and  right 
panels  represent  the  path  judgments  from  the 
simulated  gaze  rotation  condition.  As  we  have 
observed  before,  observers  perceive  curved  paths  in 
this  condition  and  thus  make  systematic  path  judgment 
errors.  The  filled  squares  in  the  left  panel  represent  the 
path  judgments  from  the  head  rotation  condition;  the 
errors  were  small,  much  as  they  were  during  real  eye 
rotations  (Royden  et  al,  1994).  This  finding  suggests 
that  some  signal  issued  during  active  head  rotations  is 
used  in  estimating  self-motion  paths.  When  the  head 
rotates  actively,  three  signals  are  created:  efferent 
signals  to  the  neck  muscles,  proprioceptive  signals 
among  the  neck  muscles,  and  signals  from  the  semi¬ 
circular  canals.  The  body  rotation  condition  was  run  to 
help  determine  which  of  those  signals  is  used:  In  a  full- 
body  rotation,  efferent  and  proprioceptive’ signals  from 
the  neck  muscles  are  not  created,  so  only  canal  signals 
are.  The  crosses  in  the  right  panel  are  the  data  from  the 
body  rotation  condition.  As  you  can  see,  large  errors 
were  observed;  indeed,  the  data  are  essentially 
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identical  to  those  in  the  simulated  gaze  rotation 
condition  (unfilled  circles).  This  finding  suggests  that 
the  presence  of  canal  signals  do  not  reduce  the  bias 
observed  during  simulated  rotations.  (A  control 
condition  showed  that  canal  signals  were 
supra  threshold.)  In  other  words,  canal  signals  do  not 
play  the  same  role  as  extra-retinal  signals  from 
rotations  of  the  eye  relative  to  the  head  and  of  the  head 
relative  to  the  body.  This  is  an  extremely  important 
finding  that  clearly  has  relevance  to  our  understanding 
of  some  illusions  experienced  by  pilots  of  high- 
performance  aircraft.  One  of  the  main  themes  in  the 
research  proposed  for  the  next  grant  period  is  to 
continue  this  line  of  investigation  into  the  use  of  the 
variety  of  extra-retinal  signals  (particularly  when  they 
conflict  as  occurs  in  aviation)  that  are  issued  during 
.self-motion. 


Head  Rotation  Body  Rotation 


Figure  17  Path  |iidgmcnt  errors  during  simulated  gaze 
rotations,  head  rotations,  and  body  rotations.  Path  errors  are 
plotted  as  a  function  of  gaze  rotation  rate.  Unfilled  circles: 
simulated  gaze  rotations.  Filled  squares:  active  head 
rotations.  C  rossos:  full-body  rotations. 

During  the  past  year,  we  also  began  to  investigate 
how  stimulation  of  the  otoliths  (the  parts  of  the 
vestibular  apparatus  that  signal  linear  acceleration) 
affects  the  perception  of  heading.  Such  investigations 
are  clearly  relevant  to  understanding  visual-vestibular 
illusions  that  occur  in  aviation  such  as  the  pitch-up 


(somatogravic)  illusion  and  the  bank  illusion  (which 
can  lead  to  the  death  spiral). 

We  presented  subjects  optic  flow  displays 
simulating  forward  translation  and  a  gaze  rotation  (see 
Figure  15B).  Normally,  observers  say  they  perceive 
curvilinear  self-motion  with  such  displays.  We  found, 
however,  if  we  rolled  observers  to  simulate  correct  or 
incorrect  centrifugal  force,  we  could  strongly  bias  their 
percepts  of  self-motion  path.  This  is  preliminary  work, 
but  its  success  shows  that  we  can  examine  the  use  of 
otolith  stimulation  in  self-motion  perception  from  optic 
flow.  This  work  has  not  been  presented  at  a  scientific 
meeting  yet. 

We  also  began  an  investigation  of  how  perspective 
transformations  affect  humans7  ability'  to  estimate  self- 
motion.  The  first  set  of  experiments  was  presented  at 
ARVO  this  year  (Sibigtroth  et  al,  2000).  The  optic  flow 
field  created  by  self-motion  through  a  rigid 
environment  is  an  important  cue  to  direction  of  self 
motion,  but  it's  not  the  only  visual  cue.  Consider,  for 
example,  the  case  when  you  walk  by  a  rectilinear 
building.  If  you  pass  to  the  left  side  of  a  comer  of  a 
building,  you  will  see  more  of  the  wall  on  the  comer's 
left  side  over  time.  If  you  pass  to  the  right  of  the  comer, 
you  will  see  more  of  the  wall  on  the  comer's  right  side 
over  time.  The  visual  system  can  take  advantage  of  this 
perspective  information  (assuming  that  the  comer  is 
indeed  a  right  angle  and  that  the  walls  are  indeed 
constructed  of  rectangles)  to  estimate  the  heading. 
Jeremy  Beer  at  Brooks  AFB  had  a  similar  insight  a  few 
years  ago  and  showed  the  people  are  sensitive  to  this 
information.  During  the  grant  period,  we  worked 
through  the  mathematics  and  showed  how  this 
perspective  transformation  information  could  be  used 
to  determine  the  direction  of  self-motion.  We  then 
conducted  some  preliminary  experiments  (presented  at 
ARVO)  that  showed  that  human  observers  do  indeed 
use  this  information  to  estimate  heading.  This  summer 
we  are  conducting  the  final  set  of  experiments  before 
writing  it  up  for  publication. 

3.  Software  Development 

We  have  spent  a  great  deal  of  effort  developing 
software  for  psychophysical  experimentation.  Thus, 
we  also  describe  the  developments  that  occurred 
during  the  grant  period.  These  include  development  of 
specialized  computer  graphics  programs,  optimized 
rendering  engines  and  tools  needed  to  generate 
displays  with  specific  spatial  and  temporal  properties. 
Stereo  3D,  texture  mapping,  high  frame  rate  animation 
and  real-time  digital  image  manipulations.  We  have 
also  developed  a  suite  of  external  device  control 
routines,  sensor  and  actuator  interfaces,  drivers,  control 
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algorithms,  feedback  loop  systems  (human  motor 
through  computer  sensory  channels)  and  low-level 
video  synchronization  tools  necessary  for  doing  real¬ 
time  psychophysics  experiments.  Almost  all  the  tools 
we  are  developing  are  in  form  of  MATLAB  shared 
libraries,  external  C  or  assembly  programs  interfaced  to 
and  called  from  MATLAB.  This  scheme  allows  us  to 
tap  into  powerful  high-level  programming  and  analysis 
features  of  MATLAB  while  we  implement  experiments 
that  require  our  low-level  tools  for  doing  real-time 
operations.  All  of  our  software  tools  are  made  available 
to  the  public  through  the  Bankslab  web  page 
fhttp://iohn.berkelev.edu/software.html).  They  are 
currently  used  by  many  labs  around  the  world.  (See  list 
below). 

BitmapTools 

BitmapTools  is  an  external  MATLAB  plugin  for 
generating  high-frame  rate  animations  (highest  refresh 
rate  possible  on  the  graphics  card /monitor).  It  allows 
for  design  and  display  of  both  static  bitmaps  and 
bitmap  movies  on  Macintosh  and  Windows  NT 
platform-.  BitmapTools  is  designed  around  one 
important  premise,  to  maximize  the  blitting  (RAM  to 
Video  Memory  transfer)  rate.  On  the  Macintosh, 
BitmapTools  takes  advantage  of  PowerPC  processor's 
pipelining  architecture  through  assembly  level 
tweaking-  On  the  PC  (NT),  high-bandwidth  blitting  is 
achieved  through  hardware-accelerated  calls 
(DirectDraw).  Almost  all  modem  graphics  cards 
contain  the  necessary  hardware  for  BitmapTools.  The 
issue  with  movie  players  in  general  is  the  unreliable 
animation  irame  rate.  In  BitmapTools,  real-time  frame 
rate  1-  guaranteed.  Under  normal  operations  (on  NT, 
with  no  ma|or  background  processes),  a  1024x768 
movie  can  plav  at  120  hz  without  missing  frames.  If  a 
frame  is  missed  for  some  reason,  exact  location  of  the 
fra  met  s)  in  t  he*  sequence  is  reported. 

OpenGLTools 

OpenGLTools  is  a  MATLAB  external  shared  library 
(compiled  me\  file)  that  incorporates  interactive  2D/3D 
graphics  lunctionality  into  MATLAB.  The  main 
objective  is  to  bridge  MATLAB's  -  high-level 
programming  environment  with  the  low-level  OpenGL 
graphics  engine.  This  is  useful  because  MATLAB's  data 
types  and  syntax  are  most  natural  for  creation  of  basic 
3D  constructs,  as  well  as  hierarchical  development  and 
manipulation  of  the  more  complex  graphics  objects. 
OpenGLTools  is  augmented  by  a  rich  collection  of 
operators  and  functions  (toolboxes)  embedded  in 
MATLAB.  It  is  designed  as  a  research  tool  for  vision 
scientists  to  create  interactive  visual  stimuli  with 
precise  control  over  spatial,  luminance  and  temporal 
properties.  Some  of  the  advanced  features  include 


stereo  (anaglyph  and  LCD  shutter  glasses),  texture 
mapping,  lighting,  buffer  manipulations,  image 
processing  filters,  and  re-programmable  interactive 
mouse  bindings.  OpenGLTools  is  available  on  MacOS, 
Windows,  and  Unix  (IRIX),  although  Windows  (NT)  is 
the  best  supported  platform. 

FlightTools 

FlightTools  is  a  flight  simulation  construction  plugin 
for  MATLAB.  Like  OpenGLTools,  it  takes  advantage  of 
hardware-accelerated  OpenGLTools  calls.  The  user  can 
define  scene  elements  in  form  of  MATLAB  matrices 
and  lists  and  specify  a  flight  path  and  camera  gaze  lists. 
Real-time  animation  of  flight  allows  interactive  control 
of  flight  parameters  such  as  pitch,  yaw  and  roll  control 
as  well  as  other  parameters  used  in  construction  of 
specific  simulation  functions. 

SerialTools 

Macintosh  serial  port  communication  driver  code 
with  an  interface  to  MATLAB. 

DAQTools 

PPC,  analog  IO  code  for  acquiring  external  analog 
signals  via  the  card  inside  the  PC.  implemented 
features  include  highest  level  synchronization  of 
internal  machine  processes  with  independent  data 
acquisition  and  processing  events  that  take  place  in  the 
IO  card.  Synchronized  signal  sampling  and  generation. 

RemoteDAQ  Tools 

A  remote  data  acquisition  system,  allowing  a 
Macintosh  to  tap  over  the  serial  lines,  into  Data 
acquisition  functions  of  an  IO  card  housed  on  a  host 
computer.  It  implements  low-level  synchronization 
features  for  guaranteeing  synchronization  of  internal 
processes  in  the  client  machine  with  various  data 
acquisition  functions  of  the  IO  card  on  the  remote  host 
machine  (ex:  synchronization  of  the  vertical  video 
blanking  signal  on  the  client  with  the  signal  sampling 
process  on  die  remote  host  computer  housing  the  signal 
acquisition  hardware).  To  give  you  an  idea  on  the 
effectiveness  of  this  tool,  it  was  the  basis  for  our  VEP 
signal  acquisition  and  processing  system  now  used  by 
Dr.  Catherine  Suttle  in  her  VEP  project  at  the 
University  of  Sydney.  It  implements  the  same 
fundamental  functionality  of  the  well-known  VEP 
system  that  took  Smith  Kettlewell  a  number  of  years  to 
develop. 

4.  Publications  during  Grant  Period 

Referred  Journals 

1.  Backus,  B.T.  &  Banks,  M.S.  (1999)  Estimator 
reliability  and  distance  scaling  in  stereoscopic 
slant  perception.  Perception,  28, 217-242. 
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2.  Backus,  B.T.,  Banks,  M.S.,  van  Ee,  R.,  &  CroweU,  J.A. 

(1999)  Horizontal  and  vertical  disparity,  eye 
position  and  stereoscopic  slant  perception.  Vision 
Research,  39,  1143-1170. 

3.  Banks,  M.S.  &  Backus,  B.T.  (1998)  Extra-retinal  and 

perspective  cues  explain  the  small  range  of  the 
induced  effect.  Vision  Research,  38, 187-194. 

4.  Crowell,  J.A.,  Banks,  M.S.,  Shenoy,  K.V.,  & 
Andersen,  R.A.  (1998)  Visual  self-motion 
perception  during  head  turns  is  mediated  by  a 
non-linear  interaction  between  three  extra-retinal 
cues.  Nature  Neuroscience,  1, 732-737. 

3.  Ehrlich,  S.M.,  Beck,  D.,  Backus,  B.T.,  Crowell,  J.A.,  & 
Banks,  M.S.  (1998)  Depth  information  and  the 
perception  of  heading.  Vision  Research,  38,  3129- 
314b. 

6.  Ernst,  M.O.,  Banks,  M.S.,  &  Biilthoff,  H.H.  (2000). 

Touch  can  change  visual  slant  perception.  Nature 
Neuroscience,  3,  69-73. 

7.  Ereeman,  T.C.A.  (1999).  Path  perception  and  Filehne 

illusion  compared:  model  and  data.  Vision 
Research,  39,  2639-266 7. 

S  Freeman,  T.C.A.  &  Banks,  M.S.  (1998)  Perceived 
speed  during  eye  movements  is  affected  by  both 
extra-retinal  and  retinal  errors.  Vision  Research,  38, 
041  -'■Mb. 

o  Freeman.  T.C.A.  Crowell,  J.A.  &  Banks,  M.S.  (1999) 
Extra-retinal  and  retinal  amplitude  and  phase 
errors  during  FileFme  illusion  and  path 
perception.  Perception  &  Psychophysics,  in  press. 

10  van  Ee.  R„  Banks,  M.S.,  &  Backus,  B.T.  (1999).  An 
anahsis  of  binocular  slant  contrast.  Perception,  28, 
1121-1 143. 

11  van  Ee.  K„  Banks,  M.S.,  &  Backus,  B.T.  (1999). 
Percened  visual  direction  near  an  occluder.  Vision 
Kc-ca-C:  39.  4083-4097. 

Chapters. 

12.  Banks.  M.S.  &  Backus,  B.T.  (1998)  Use  of  horizontal 
disparitv.  vertical  disparity,  and  eye  position  in 
slant  perception.  In  L.  Harris  (ed.).  Vision  and 
Action.  Oxford  University  Press. 

Manuscripts  under  Review  or  in  Preparation 

13.  Adams,  W.J.  Adams,  Banks,  M.S.,  &  van  Ee,  R. 
(2000)  Adaptation  to  3d  distortions  in  human 
vision.  Nature  Neuroscience,  in  revision. 

14.  Banks,  M.  S.,  Hooge,  I.  T.  C.,  &  Backus,  B.  T.  (2000) 
Horizontal  and  vertical  disparities,  torsion  signals, 
and  perception  of  inclined  surfaces.  Vision 
Research. 

13.  Domini,  F.,  Adams,  W.J.,  &  Banks,  M.S.  (2000)  3d 
aftereffects  are  due  to  shape  and  not.  disparity 


adaptation.  Vision  Research,  in  revision. 

16.  James,  F.M.K.,  Humphrey,  G.K.,  Banks,  M.S.,  & 
Vilis,  T.  (2000).  Accurate  slant  judgements  bsed  on 
extra-retinal  eye  position.  Vision  Research,  in 
preparation. 

17.  Hillis,  J.M.  &  Banks,  M.S.  (20001.  Are 
corresponding  points  fixed?  Vision  Research,  in 
preparation. 

18.  Hooge,  I.T.C.,  Banks,  M.S.,  &  van  den  Berg,  A.V. 
(2000).  Subjective  and  objective  measures  of 
cyclovergence.  Vision  Research,  in  preparation. 

5.  Service  for  Air  Force 

During  the  grant  period,  the  PI  was  asked  to  do  a 
few  things  that  might  potentially  benefit  the  Air  Force. 

In  1998,  he  traveled  to  Williams  AFB  in  Arizona  in 
order  to  meet  with  Byron  Pierce  and  George  Geri. 
During  that  trip,  he  consulted  with  Drs.  Pierce  and  Geri 
on  their  ongoing  research  and  discussed  possible 
collaborations  between  the  Berkeley  and  Williams'  labs. 
This  led  to  an  equipment  loan  in  which  Williams  sent 
us  an  SGI  Crimson  workstation  and  a  Sony  CRT 
projector. 

In  2000,  the  PI  was  asked  to  join  a  team  that  would 
put  together  a  research  plan  for  the  Spatial 
Disorientation  Program  for  the  Air  Force  Research 
Labs.  This  work  consisted  of  reviewing  the  previous 
research  plans,  evaluating  a  plan  written  by 
investigators  at  Wright-Patterson  AFB,  and  then 
traveling  to  Brooks  AFB  for  a  two-day  meeting  chaired 
by  Bill  Ercoline.  The  outcome  was  a  5-year  research 
plan  that  is  currently  being  evaluated  by  the  Air  Force. 

In  2000,  the  PI  participated  in  the  Civic  Outreach 
Program  for  two  days.  He  traveled  to  Moffett  Airfield, 
Edwards  AFB,  Cheyenne  Mountain,  and  Peterson  AFB 
and  participated  in  briefings,  tours,  and  discussions 
with  Air  Force  personnel. 

6.  Significance  of  Research  Program 
for  Air  Force 

During  the  past  grant  period,  we  examined  visual 
navigation  and  space  perception  in  humans.  We 
believe  that  our  research  is  highly  relevant  to  the 
military  aviation  mission.  The  main  area  of  Air  Force 
need  that  is  addressed  by  our  research  is  spatial 
disorientation  (SD)  and  the  use  of  synthetic  or 
enhanced  visual  display  devices  such  as  head-mounted 
displays  (HMDs),  night-vision  goggles  (NVGs),  the 
advanced  aircraft  control  station  (ACS),  and  more. 

SD  remains  a  major  safety  problem  in  flight  and  SD 
is  likely  to  become  an  even  more  serious  problem  as  the 
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next  wave  of  aircraft  (e.g.,  agile  flight)  is  developed  and 
put  into  flight.  Our  work  on  heading  perception  is 
aimed  at  determining  the  complex  of  visual  and  non¬ 
visual  signals  that  are  used  to  estimate  the  direction  of 
self-motion  and  ones  orientation  with  respect  to 
gravity.  Specifically,  we  are  working  on  determining 
how  much  weight  is  given  to  various  signals  (e.g.,  optic 
flow  vs  vestibular)  and  how  those  weights  depend  on 
the  viewing  situation  (e.g.,  weight  given  to  vestibular 
increases  as  the  optic  flow  information  is  degraded). 
With  a  better  understanding  of  how  the  human 
nervous  system  computes  and  weights  these  various 
sources  of  information  we  will  be  able  to  provide  the 
Air  Force  material  relevant  to  pilot  training,  cockpit 
design,  and  the  configuration  of  synthetic  visual 
displays.  Let  us  give  one  specific  example.  In  our  work 
on  the  somatogravic  ("pitch  up")  illusion,  we  are  trying 
to  determine  what  visual  cues  must  be  present  in  order 
to  override  the  vestibular-based  illusion  of  upward 
pitch.  Once  we  know  what  the  critical  visual  cues  are, 
we  can  recommend  the  design  of  an  artificial  cockpit 
display  te  g.,  an  artificial  horizon)  that  would  minimize 
the  illusion. 

Our  work  on  space  perception,  primarily  slant  and 
curvature  perception,  is  also  quite  relevant  to  the 
military  aviation  mission.  In  the  next  generation  of 
militarv  aircraft,  we  will  see  greater  and  greater 
reliance  on  synthetic  visual  displays.  Indeed,  if  the 
closed  cockpit  (all  virtual)  aircraft  is  brought  on  line,  all 
ol  the  visual  information  provided  to  the  pilot  will  be 
synthetic.  We  have  found  that  perceived  depth  is  based 
on  the  integration  of  numerous  visual  (e.g.,  disparity 
and  texture  gradient)  and  non-visual  (e.g.,  eye  muscle 
signals'  cues.  The  Itnal  percept  is  the  result  of  a 
weighted  combination  of  those  various  cues.  An 
understanding  of  how  those  cues  are  calculated  and 
weighted  in  the  nervous  system  is  critical  to  the  design 
ot  a  synthetic  visual  display.  For  example,  we  know 
trorn  our  work  that  cues  provided  by  the  CRT  itself 
(e  g„  pixeh/ation,  focus  cues)  cause  perceptual  depth 
compression  Such  compression  would  be  highly 
undesirable  in  an  all-virtual  cockpit  and  so  the  design 
of  the  visual  display  will  either  have  to-reduce  the 
influence  of  such  competing  cues  or  figure  out  how  to 
override  them. 

Finally,  our  software  development  might  also  be 
quite  useful  to  the  Air  Force.  At  a  meeting  at  Brooks 
AFB  in  March,  2000  (chaired  by  Bill  Ercoline),  a 
potential  business  plan  was  formed  for  the  next  5  years. 
One  idea  presented  in  this  plan  was  to  generate  web- 
based  instructional  aides  for  teaching  spatial 
disorientation  to  future  pilots.  One  of  our  software 
developments  —  FlightTools  —  would  allow  us  to 
recreate  flight  scenarios  that  could  be  played  on  the 


internet  for  use  in  the  classroom.  Those  scenarios  could 
be  seen  from  the  perspective  of  the  pilot  or  from  an 
outside  perspective  (chosen  by  the  student).  The 
scenarios  can  be  produced  on  standard  PCs  with  off- 
the-shelf  video  cards.  The  Banks  Lab  is  committed 
through  its  relationship  with  the  Air  Force  to  produce 
material  like  this  whenever  it  might  be  needed. 
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